Skip to main content

Reply to this post | See parent post | Go Back
View Post [edit]

Poster: Anamon Date: Jul 4, 2012 5:56pm
Forum: web Subject: Re: Do crawls get deleted?

There are several reasons why that could happen. Since archived data is distributed across many servers, it is possible that there is a problem with one of them. I think usually the crawl will still be shown in the graph, but a message will be displayed when you try to open it. It is possible that some other failure can lead to the crawl not being shown on the overview, or that maybe the failure is permanent (e.g. hard disk crash) and the crawl will not return.

There's also the possibility that the crawl in question was requested to be removed. This would be especially likely if it falls into a time period where the address has belonged to a different entity than during the time periods where crawls are still available.

Generally, crawls don't get deleted just like that, but you have to keep in mind that there's always the possibility of them disappearing forever.

As a more hands-on solution. you could try the Wayback Machine at the Bibliotheca Alexandrina ( They run separately, so the crawls either of them have are independent of each other.

Reply to this post
Reply [edit]

Poster: KlangenFarben Date: Jul 6, 2012 6:41pm
Forum: web Subject: Re: Do crawls get deleted?

I can certainly attest to the phenomena of previously available TWBM crawls that became unavailable sometime in the past year or two. I launched in late November 2008, and shortly thereafter subscribed to's crawl list. By 2010 there were at least three crawls, but currently there's a single crawl that's sixteen months old; not much has changed since then (mostly deletions), suggesting some heuristic was at work. I'm not challenging Anamon's account but rather relating my perception of how TWBM works.

Anamon second paragraph does accurately reflect what happened to the once-well-crawled, which I operated for three years or so 1997-1999 and perhaps later. When I first heard of TWBM around 2004-2006, I immediately checked and was astonished to find at least three and perhaps as many as six crawls. The removal request is clear, but as I am a data hoarder I posted a copy of the old site on my current site, so--outside of loss of proof of previous ownership--no harm done, and clearly no complaints as I've never been asked much less assisted with TWBM.

At any rate, it's Good To Know some of the inner workings of TWBM.

cheers klang